Skip to content

fix: Correctly propagate user arguments to Spark connect server#696

Merged
sbernauer merged 3 commits into
mainfrom
fix/connect-user-argument-list
Jun 11, 2026
Merged

fix: Correctly propagate user arguments to Spark connect server#696
sbernauer merged 3 commits into
mainfrom
fix/connect-user-argument-list

Conversation

@sbernauer

@sbernauer sbernauer commented Jun 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

A customer asked me how to do Iceberg with a SparkConnect server. So I tried and noticed this bug.
Added a kuttl test as well.

From

  - args:                                                                                                                                                                                                          
    - |-                                                                                                                                                                                                           
      containerdebug --output=/stackable/log/containerdebug-state.json --loop &                                                                                                                                    
                                                                                                                                                                                                                   
      cp /stackable/spark/conf/spark-defaults.conf /tmp/spark.properties                                                                                                                                           
      config-utils template /tmp/spark.properties                                                                                                                                                                  
                                                                                                                                                                                                                   
      /stackable/spark/sbin/start-connect-server.sh \                                                                                                                                                              
      --deploy-mode client \                                                                                                                                                                                       
      --master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS} \                                                                                                                         
      --properties-file /tmp/spark.properties                                                                                                                                                                      
       --package org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0

to

  - args:                                                                                                                                                                                                          
    - |-                                                                                                                                                                                                           
      containerdebug --output=/stackable/log/containerdebug-state.json --loop &                                                                                                                                    
                                                                                                                                                                                                                   
      cp /stackable/spark/conf/spark-defaults.conf /tmp/spark.properties                                                                                                                                           
      config-utils template /tmp/spark.properties                                                                                                                                                                  
                                                                                                                                                                                                                   
      /stackable/spark/sbin/start-connect-server.sh \                                                                                                                                                              
      --deploy-mode client \                                                                                                                                                                                       
      --master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS} \                                                                                                                         
      --properties-file /tmp/spark.properties \                                                                                                                                                                    
      --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@sbernauer sbernauer marked this pull request as ready for review June 10, 2026 12:27
@sbernauer sbernauer requested a review from razvan June 10, 2026 12:27
@sbernauer sbernauer self-assigned this Jun 10, 2026
@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jun 10, 2026
@razvan razvan moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jun 10, 2026
razvan
razvan approved these changes Jun 10, 2026
View reviewed changes

@razvan razvan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@sbernauer sbernauer added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Jun 11, 2026
@sbernauer sbernauer changed the title fix: Correctly propage user arguments to Spark connect server fix: Correctly propagate user arguments to Spark connect server Jun 11, 2026
@sbernauer

sbernauer commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Release notes

Fixed

Correctly propagate user arguments to Spark connect server

@sbernauer sbernauer requested a review from razvan June 11, 2026 06:18
razvan
razvan approved these changes Jun 11, 2026
View reviewed changes

@razvan razvan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@sbernauer

sbernauer commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Started https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/74/

@sbernauer

sbernauer commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

The two failed tests passed locally

@sbernauer sbernauer added this pull request to the merge queue Jun 11, 2026
@sbernauer sbernauer moved this from Development: In Review to Development: Done in Stackable Engineering Jun 11, 2026
Merged via the queue into main with commit cb5467f Jun 11, 2026
12 checks passed
@sbernauer sbernauer deleted the fix/connect-user-argument-list branch June 11, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@razvan razvan razvan approved these changes

Assignees

@sbernauer sbernauer

Labels

customer-request release/26.7.0 release-note Denotes a PR that will be considered when it comes time to generate release notes. type/bug

Projects

Stackable Engineering
Status: Development: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sbernauer @razvan @stackable-bot